Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(zetaclient): subscribe to new blocks in scheduler #3228

Merged
merged 6 commits into from
Dec 10, 2024

Conversation

gartnera
Copy link
Member

@gartnera gartnera commented Nov 27, 2024

Use cometbft block subscription to improve signer synchronization.

Closes #3227

Performance tests show ~25% improvement in overall runtime and massive reductions p50+ stats:

Before:

perf_eth_wit | Latency report:
perf_eth_wit | min:  11.52
perf_eth_wit | max:  183.42
perf_eth_wit | mean: 84.64
perf_eth_wit | std:  48.25
perf_eth_wit | p50:  76.64
perf_eth_wit | p75:  120.73
perf_eth_wit | p90:  157.83
perf_eth_wit | p95:  171.35
perf_eth_wit | all withdraws completed
perf_eth_wit | ✅ completed in 6m2.423686072s - stress test Ether withdrawal

After:

perf_eth_wit | Latency report:
perf_eth_wit | min:  9.03
perf_eth_wit | max:  13.04
perf_eth_wit | mean: 11.60
perf_eth_wit | std:  1.20
perf_eth_wit | p50:  12.03
perf_eth_wit | p75:  12.53
perf_eth_wit | p90:  12.54
perf_eth_wit | p95:  13.03
perf_eth_wit | all withdraws completed
perf_eth_wit | ✅ completed in 4m26.799384212s - stress test Ether withdrawal

I expect improvements on mainnet and testnet but not as extreme without some additional latency correction.

Summary by CodeRabbit

Release Notes

  • New Features

    • Integrated CometBFT client for enhanced transaction scheduling and event handling.
    • Updated orchestration process to react to new block events from CometBFT.
  • Bug Fixes

    • Improved error handling for WebSocket events and block event subscriptions.
  • Tests

    • Enhanced testing framework with mock CometBFT client for better simulation of interactions and error scenarios.

@gartnera gartnera added the no-changelog Skip changelog CI check label Nov 27, 2024
Copy link
Contributor

coderabbitai bot commented Nov 27, 2024

Important

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

📝 Walkthrough
📝 Walkthrough
📝 Walkthrough
📝 Walkthrough
📝 Walkthrough
📝 Walkthrough
📝 Walkthrough
📝 Walkthrough
📝 Walkthrough
📝 Walkthrough
📝 Walkthrough
📝 Walkthrough
📝 Walkthrough

Walkthrough

The pull request introduces significant enhancements to the zetaclientd process by integrating the CometBFT client. Key modifications include the establishment of a CometBFT client connection, updates to the orchestration logic to utilize event-driven block subscriptions instead of polling, and improved error handling throughout the codebase. The changes also extend to the testing framework, incorporating mock clients for better validation of interactions with the CometBFT RPC layer. Overall, the modifications aim to improve synchronization and responsiveness within the zetaclient architecture.

Changes

File Path Change Summary
cmd/zetaclientd/start.go Added import for CometBFT HTTP client, updated Start function to create CometBFT client, enhanced error handling for WebSocket events.
zetaclient/orchestrator/orchestrator.go Added CometBFT client field to Orchestrator struct, refactored runScheduler to use event-driven block subscriptions, improved error handling.
zetaclient/orchestrator/orchestrator_test.go Enhanced mockOrchestrator with CometBFT mock client, updated tests for signers and observers, added error handling tests.

Assessment against linked issues

Objective Addressed Explanation
Subscribe to new block events via websockets (#3227)

Possibly related PRs

Suggested labels

SOLANA_TESTS

Suggested reviewers

  • fbac
  • kingpinXD
  • ws4charlie
  • lumtis
  • skosito
  • brewmaster012
  • swift1337

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

❤️ Share
🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai generate docstrings to generate docstrings for this PR. (Experiment)
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

Copy link

!!!WARNING!!!
nosec detected in the following files: zetaclient/orchestrator/orchestrator.go

Be very careful about using #nosec in code. It can be a quick way to suppress security warnings and move forward with development, it should be employed with caution. Suppressing warnings with #nosec can hide potentially serious vulnerabilities. Only use #nosec when you're absolutely certain that the security issue is either a false positive or has been mitigated in another way.

Only suppress a single rule (or a specific set of rules) within a section of code, while continuing to scan for other problems. To do this, you can list the rule(s) to be suppressed within the #nosec annotation, e.g: /* #nosec G401 */ or //#nosec G201 G202 G203
Broad #nosec annotations should be avoided, as they can hide other vulnerabilities. The CI will block you from merging this PR until you remove #nosec annotations that do not target specific rules.

Pay extra attention to the way #nosec is being used in the files listed above.

@github-actions github-actions bot added the nosec label Nov 27, 2024
@gartnera gartnera force-pushed the zetaclient-subscribe-new-blocks branch 2 times, most recently from 5733a5b to 276e5ad Compare November 27, 2024 20:25
Copy link

codecov bot commented Nov 27, 2024

Codecov Report

Attention: Patch coverage is 14.28571% with 102 lines in your changes missing coverage. Please review.

Project coverage is 61.81%. Comparing base (3a8fa1e) to head (365121f).
Report is 1 commits behind head on develop.

Files with missing lines Patch % Lines
zetaclient/orchestrator/orchestrator.go 0.00% 81 Missing ⚠️
zetaclient/zetacore/client.go 33.33% 11 Missing and 1 partial ⚠️
zetaclient/zetacore/client_subscriptions.go 57.89% 6 Missing and 2 partials ⚠️
zetaclient/zetacore/client_cosmos.go 0.00% 1 Missing ⚠️
Additional details and impacted files

Impacted file tree graph

@@             Coverage Diff             @@
##           develop    #3228      +/-   ##
===========================================
- Coverage    61.84%   61.81%   -0.04%     
===========================================
  Files          431      432       +1     
  Lines        30848    30875      +27     
===========================================
+ Hits         19077    19084       +7     
- Misses       10913    10930      +17     
- Partials       858      861       +3     
Files with missing lines Coverage Δ
zetaclient/metrics/metrics.go 75.00% <ø> (ø)
zetaclient/zetacore/client_cosmos.go 44.44% <0.00%> (+10.15%) ⬆️
zetaclient/zetacore/client_subscriptions.go 57.89% <57.89%> (ø)
zetaclient/zetacore/client.go 57.89% <33.33%> (-8.14%) ⬇️
zetaclient/orchestrator/orchestrator.go 18.45% <0.00%> (-0.04%) ⬇️

@gartnera gartnera added the PERFORMANCE_TESTS Run make start-e2e-performance-test label Nov 27, 2024
@gartnera gartnera marked this pull request as ready for review November 27, 2024 21:49
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

🧹 Outside diff range and nitpick comments (7)
cmd/zetaclientd/start.go (1)

Line range hint 254-266: Consider addressing the disabled zeta supply checker

The TODO comment indicates that the zeta supply checker is disabled due to performance concerns with GRPC queries. This might be worth revisiting now that we're improving synchronization performance.

Would you like me to:

  1. Open a GitHub issue to track the re-enablement of the supply checker?
  2. Propose an implementation that addresses the performance concerns?
zetaclient/orchestrator/orchestrator_test.go (1)

591-591: Consider enhancing mock client initialization.

While the mock client is correctly added to the Orchestrator struct, the current initialization could be improved to better support testing different scenarios.

Consider this enhancement:

-		cometbftClient: &cometbft_rpc_client_mock.Client{},
+		cometbftClient: func() *cometbft_rpc_client_mock.Client {
+			mockClient := &cometbft_rpc_client_mock.Client{}
+			// Initialize with default expectations
+			mockClient.On("Start").Return(nil)
+			return mockClient
+		}(),
zetaclient/orchestrator/orchestrator.go (5)

321-330: Simplify Operator Balance Update Logic

The current logic for updating the operator balance includes unnecessary checks and can be streamlined for clarity and efficiency.

diff --git a/zetaclient/orchestrator/orchestrator.go b/zetaclient/orchestrator/orchestrator.go
--- a/zetaclient/orchestrator/orchestrator.go
+++ b/zetaclient/orchestrator/orchestrator.go
@@ -324,13 +324,9 @@
 } else {
     diff := oc.lastOperatorBalance.Sub(balance)
-    if diff.GT(sdkmath.NewInt(0)) && diff.LT(sdkmath.NewInt(math.MaxInt64)) {
+    if diff.IsPositive() {
         oc.ts.AddFeeEntry(bn, diff.Int64())
         oc.lastOperatorBalance = balance
     }
 }

341-344: Enhance Error Logging for Clarity

Improve the error message when failing to retrieve pending cross-chain transactions to provide clearer context for troubleshooting.

-oc.logger.Error().Err(err).Msgf("runScheduler: GetPendingCctxsWithinRatelimit failed")
+oc.logger.Error().Err(err).Msg("failed to retrieve pending cross-chain transactions within rate limit")
🧰 Tools
🪛 GitHub Check: codecov/patch

[warning] 341-344: zetaclient/orchestrator/orchestrator.go#L341-L344
Added lines #L341 - L344 were not covered by tests


358-360: Provide Additional Context in Error Messages

When logging errors during signer resolution, include the chain ID and relevant details to aid in debugging.

 oc.logger.Error().Err(err).
     Int64(logs.FieldChain, chainID).
-    Msg("runScheduler: unable to resolve signer")
+    Msgf("unable to resolve signer for chain ID %d", chainID)
🧰 Tools
🪛 GitHub Check: codecov/patch

[warning] 358-360: zetaclient/orchestrator/orchestrator.go#L358-L360
Added lines #L358 - L360 were not covered by tests


364-369: Avoid Redundant Error Handling in Observer Resolution

The error handling logic duplicates context provided by the errors.Wrapf function. Streamlining this can enhance readability.

 if err != nil {
-    oc.logger.Error().Err(err).
-        Int64(logs.FieldChain, chainID).
-        Msg("runScheduler: unable to resolve observer")
+    oc.logger.Error().Err(err).Msgf("unable to resolve observer for chain ID %d", chainID)
     continue
 }
🧰 Tools
🪛 GitHub Check: codecov/patch

[warning] 364-369: zetaclient/orchestrator/orchestrator.go#L364-L369
Added lines #L364 - L369 were not covered by tests


299-406: Increase Test Coverage for New Scheduler Logic

The new event-driven scheduler implementation lacks test coverage, as indicated by static analysis. Enhancing tests will improve reliability and prevent regressions.

Would you like me to assist in creating unit tests for the runScheduler method to ensure thorough coverage?

🧰 Tools
🪛 GitHub Check: codecov/patch

[warning] 299-302: zetaclient/orchestrator/orchestrator.go#L299-L302
Added lines #L299 - L302 were not covered by tests


[warning] 309-329: zetaclient/orchestrator/orchestrator.go#L309-L329
Added lines #L309 - L329 were not covered by tests


[warning] 333-338: zetaclient/orchestrator/orchestrator.go#L333-L338
Added lines #L333 - L338 were not covered by tests


[warning] 341-344: zetaclient/orchestrator/orchestrator.go#L341-L344
Added lines #L341 - L344 were not covered by tests


[warning] 347-350: zetaclient/orchestrator/orchestrator.go#L347-L350
Added lines #L347 - L350 were not covered by tests


[warning] 353-356: zetaclient/orchestrator/orchestrator.go#L353-L356
Added lines #L353 - L356 were not covered by tests


[warning] 358-360: zetaclient/orchestrator/orchestrator.go#L358-L360
Added lines #L358 - L360 were not covered by tests


[warning] 364-369: zetaclient/orchestrator/orchestrator.go#L364-L369
Added lines #L364 - L369 were not covered by tests


[warning] 373-379: zetaclient/orchestrator/orchestrator.go#L373-L379
Added lines #L373 - L379 were not covered by tests


[warning] 383-384: zetaclient/orchestrator/orchestrator.go#L383-L384
Added lines #L383 - L384 were not covered by tests


[warning] 388-401: zetaclient/orchestrator/orchestrator.go#L388-L401
Added lines #L388 - L401 were not covered by tests


[warning] 406-406: zetaclient/orchestrator/orchestrator.go#L406
Added line #L406 was not covered by tests

📜 Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL

📥 Commits

Reviewing files that changed from the base of the PR and between 08ff881 and 276e5ad.

📒 Files selected for processing (3)
  • cmd/zetaclientd/start.go (3 hunks)
  • zetaclient/orchestrator/orchestrator.go (5 hunks)
  • zetaclient/orchestrator/orchestrator_test.go (2 hunks)
🧰 Additional context used
📓 Path-based instructions (3)
cmd/zetaclientd/start.go (1)

Pattern **/*.go: Review the Go code, point out issues relative to principles of clean code, expressiveness, and performance.

zetaclient/orchestrator/orchestrator.go (1)

Pattern **/*.go: Review the Go code, point out issues relative to principles of clean code, expressiveness, and performance.

zetaclient/orchestrator/orchestrator_test.go (1)

Pattern **/*.go: Review the Go code, point out issues relative to principles of clean code, expressiveness, and performance.

🪛 GitHub Check: codecov/patch
zetaclient/orchestrator/orchestrator.go

[warning] 114-114: zetaclient/orchestrator/orchestrator.go#L114
Added line #L114 was not covered by tests


[warning] 299-302: zetaclient/orchestrator/orchestrator.go#L299-L302
Added lines #L299 - L302 were not covered by tests


[warning] 309-329: zetaclient/orchestrator/orchestrator.go#L309-L329
Added lines #L309 - L329 were not covered by tests


[warning] 333-338: zetaclient/orchestrator/orchestrator.go#L333-L338
Added lines #L333 - L338 were not covered by tests


[warning] 341-344: zetaclient/orchestrator/orchestrator.go#L341-L344
Added lines #L341 - L344 were not covered by tests


[warning] 347-350: zetaclient/orchestrator/orchestrator.go#L347-L350
Added lines #L347 - L350 were not covered by tests


[warning] 353-356: zetaclient/orchestrator/orchestrator.go#L353-L356
Added lines #L353 - L356 were not covered by tests


[warning] 358-360: zetaclient/orchestrator/orchestrator.go#L358-L360
Added lines #L358 - L360 were not covered by tests


[warning] 364-369: zetaclient/orchestrator/orchestrator.go#L364-L369
Added lines #L364 - L369 were not covered by tests


[warning] 373-379: zetaclient/orchestrator/orchestrator.go#L373-L379
Added lines #L373 - L379 were not covered by tests


[warning] 383-384: zetaclient/orchestrator/orchestrator.go#L383-L384
Added lines #L383 - L384 were not covered by tests


[warning] 388-401: zetaclient/orchestrator/orchestrator.go#L388-L401
Added lines #L388 - L401 were not covered by tests


[warning] 406-406: zetaclient/orchestrator/orchestrator.go#L406
Added line #L406 was not covered by tests

🔇 Additional comments (5)
cmd/zetaclientd/start.go (1)

239-239: Verify orchestrator's usage of CometBFT client

The integration of CometBFT client with the orchestrator looks correct. However, let's verify its usage in the orchestrator package.

✅ Verification successful

CometBFT client integration verified and properly implemented

The orchestrator correctly uses the CometBFT client for block subscription:

  • Client is properly defined in the Orchestrator struct (cometbftClient cometbft_rpc_client.Client)
  • Block subscription is implemented in zetaclient/orchestrator/orchestrator.go using Subscribe method with tm.event='NewBlock'
  • The subscription is used to receive new block events through blockEventChan
🏁 Scripts executed

The following scripts were executed for the analysis:

Script:

#!/bin/bash
# Description: Verify how the CometBFT client is used in the orchestrator package

# Check for block subscription usage
ast-grep --pattern 'func (o *Orchestrator) runScheduler() {
  $$$
  Subscribe($$$)
  $$$
}'

# Check for proper cleanup
ast-grep --pattern 'func (o *Orchestrator) Stop() {
  $$$
  Stop()
  $$$
}'

Length of output: 171


Script:

#!/bin/bash
# Let's try a broader search to understand the orchestrator's implementation

# Find orchestrator files
fd orchestrator

# Check how CometBFT client is used in orchestrator
rg -A 5 "CometClient|cometClient" 

# Look for block subscription patterns
rg -A 5 "Subscribe|EventsClient|BlockClient"

# Check orchestrator struct definition
ast-grep --pattern 'type Orchestrator struct {
  $$$
}'

Length of output: 16113

zetaclient/orchestrator/orchestrator_test.go (2)

15-15: LGTM: Import addition aligns with the new CometBFT integration.

The addition of the CometBFT RPC client mock import is consistent with the PR's objective of implementing block subscription.


Line range hint 1-675: Verify test coverage for CometBFT integration.

The test file has comprehensive coverage for existing functionality, but we should ensure adequate testing of the new CometBFT integration.

zetaclient/orchestrator/orchestrator.go (2)

333-334: Verify Accuracy of Hot Key Burn Rate Metric

Ensure that the HotKeyBurnRate metric reflects the most recent operator balance to maintain accurate monitoring.

Would you like assistance in implementing a verification method to confirm the metric's accuracy?

🧰 Tools
🪛 GitHub Check: codecov/patch

[warning] 333-338: zetaclient/orchestrator/orchestrator.go#L333-L338
Added lines #L333 - L338 were not covered by tests


373-379: Optimize Metric Updates for Pending Transactions

Updating Prometheus metrics inside the main loop may impact performance. Consider batching metric updates or moving them outside critical execution paths.

[performance]

🧰 Tools
🪛 GitHub Check: codecov/patch

[warning] 373-379: zetaclient/orchestrator/orchestrator.go#L373-L379
Added lines #L373 - L379 were not covered by tests

cmd/zetaclientd/start.go Outdated Show resolved Hide resolved
zetaclient/orchestrator/orchestrator.go Show resolved Hide resolved
zetaclient/orchestrator/orchestrator.go Outdated Show resolved Hide resolved
zetaclient/orchestrator/orchestrator.go Outdated Show resolved Hide resolved
cmd/zetaclientd/start.go Outdated Show resolved Hide resolved
Copy link
Member

@lumtis lumtis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ws4charlie @swift1337 please have a look

cmd/zetaclientd/start.go Outdated Show resolved Hide resolved
cmd/zetaclientd/start.go Outdated Show resolved Hide resolved
cmd/zetaclientd/start.go Outdated Show resolved Hide resolved
zetaclient/orchestrator/orchestrator.go Outdated Show resolved Hide resolved
zetaclient/orchestrator/orchestrator.go Outdated Show resolved Hide resolved
zetaclient/orchestrator/orchestrator.go Show resolved Hide resolved
@gartnera gartnera force-pushed the zetaclient-subscribe-new-blocks branch from 276e5ad to b6ddda8 Compare December 5, 2024 20:20
@gartnera gartnera force-pushed the zetaclient-subscribe-new-blocks branch from b6ddda8 to b19de8b Compare December 5, 2024 20:22
@gartnera gartnera removed the request for review from fbac December 5, 2024 20:34
@gartnera
Copy link
Member Author

gartnera commented Dec 5, 2024

Ok this is ready for another round of reviews. There is still some more cleanup that could be done in zetaclient regarding the tendermint/cometbft naming and multiple instantiation of cometbft rpc clients but I would prefer to do that separately.

@gartnera gartnera force-pushed the zetaclient-subscribe-new-blocks branch 3 times, most recently from f004f99 to 962c490 Compare December 5, 2024 21:41
@gartnera gartnera requested a review from a team as a code owner December 5, 2024 22:13
@gartnera gartnera force-pushed the zetaclient-subscribe-new-blocks branch from 5ae8b81 to fe86b8a Compare December 5, 2024 22:16
Copy link
Contributor

@ws4charlie ws4charlie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good overall. minor comment

@gartnera gartnera force-pushed the zetaclient-subscribe-new-blocks branch from c5a6e73 to 365121f Compare December 10, 2024 19:09
@gartnera gartnera added this pull request to the merge queue Dec 10, 2024
Merged via the queue into develop with commit 376b714 Dec 10, 2024
39 of 40 checks passed
@gartnera gartnera deleted the zetaclient-subscribe-new-blocks branch December 10, 2024 19:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
breaking:cli no-changelog Skip changelog CI check nosec PERFORMANCE_TESTS Run make start-e2e-performance-test
Projects
None yet
Development

Successfully merging this pull request may close these issues.

zetaclient scheduler becomes desynced
5 participants